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Module  Topics 


The  goal  of  performance  management 
What  is  a  “performance  measure”? 
What  do  we  measure? 

Step  through  of  review  comments 
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Motivation 


Accountability  for  performance  has  become  increasingly  critical. 

■  It  is  necessary  for  VA  to  understand  the  key  drivers  behind  its  performance 
and  demonstrate  the  value  of  its  work. 

■  Results  provide  information  on  how  VA’s  current  programs  and  activities  are 
working  and  how  resources  can  be  allocated  to  optimize  effectiveness. 

■  Focus  on  the  use  of  data  is  a  key  principle  guiding  any  performance 
management  system. 
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But  First  You  Need  to  Determine  Where  You  Are 


In  the  absence  of  measurement, 
we  are  like  a  traveler  without  a 
map  ...  wandering, 
directionless,  and  confused. 
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Measurement  helps  us  establish  where  we  are,  whether  we’ve 
made  progress,  what  has  changed,  and  where  we  need  to  go. 
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You  Can’t  Manage  What  You  Can’t  Measure 


Performance  management 


A  forward-looking  process  used  to  set  goals 
and  regularly  check  progress  toward 
achieving  those  goals. 


Performance  measurement 


A  process  by  which  an  organization  monitors 
important  aspects  of  its  programs,  systems, 
and  processes. 


Performance  measure 


The  quantitative  measure  that  is  designed 
to  track  progress  towards  achieving  a  goal. 


MOkSUteMCHT 


tcrcrzT 
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Without  Performance  Management 


Projects 
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Building  Alignment  and  Synergy 

Performance  management  strives  to  align 

all  the  VA’s  subsystems  to  achieve  results.  Mature  Performance  Management  System 

■  Divisions 

■  Processes 

■  Teams 

■  Individuals 

Transitioning  to  Performance  Management 


Without  Performance  Management 
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Cascading  and  Linked  Goals 


Increasingly 

strategic 


Increasingly 

detailed 
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OIA  Performance  Management 


OIA  Goals 


Program  Initiative 


Project 


Each  Program  Initiative  maps 
to  at  least  one  of  the  OIA  goals. 

Accomplishing  a  Program 
Initiative  supports  the 
achievement  of  an  OIA  goal(s). 


Project  Activities 
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OIA  Performance  Management 


OIA  Goals 

Program  Initiative  ^ 

A  Program  establishes  a 

Project  to  address  the  intent 
*  of  the  Program’s  Initiative. 

Project  -- 

If  the  project  is  successful,  then 
^  the  intent  of  the  initiative  is  met. 

If  the  intent  of  the  initiative  is  met, 
then  progress  is  made  in  achieving 
the  OIA  goal  that  the  initiative 
is  mapped  to. 
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OIA  Performance  Management 


OIA  Goals 


Program  Initiative 

Project 

Project  Activities 


Outcome  #1 

Outcome  #2 

Outcome  #n 

It  is  at  the  project  level  when 
work  is  actually  accomplished. 

A  project  is  composed  of 
activities. 

Activities  have  outcome(s). 

Outcomes  are  what  we 
measure. 

We  also  measure  progress 
towards  achieving  the  outcome. 
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Module  Topics 


The  goal  of  performance  management 
►  What  is  a  “performance  measure”? 
What  do  we  measure? 

Step  through  of  review  comments 
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Two  Types  of  Measures 


Quantitative 


Qualitative 
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Quantitative  Performance  Measures 


A  quantitative  performance  measure  is  composed  of 

•  a  number 

•  a  unit  of  scale 

The  number  provides  the  magnitude  (how  much) 

The  unit  of  scale  gives  the  number  a  meaning  (what). 

In  some  cases,  multi-dimensional  units  of  measure  are  used. 

•  Expressed  as  ratios. 

•  Examples:  Number  of  accidents  per  year;  #  of  certifications  awarded  per  #  of 
individuals  trained. 
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Clarifying  Some  Terminology 


Term 

Explanation 

Activity 

A  description  of  the  task  that 
produces  the  intended  outcome. 

Outcome 

The  objective  of  the  activity;  a 
product  or  service. 

Performance  measure 

Quantitative:  A  number  that 
portrays  a  key  characteristic  of 
the  outcome. 

Qualitative:  The  assignment  of  a 
categorical  variable  that 
characterizes  a  key  attribute  of 
the  outcome. 
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Clarifying  Some  Terminology 


Activity 

A  description  of  the  task  that  produces  the 
intended  outcome. 

Outcome 

The  objective  of  the  activity;  a  product  or  service. 

Performance 

measure 

Quantitative:  A  number  that  portrays  a  key 
characteristic  of  the  outcome. 

Qualitative:  The  assignment  of  a  categorical  variable 
that  portrays  a  key  characteristic  of  the 
outcome. 

Target 

A  specific  value  of  the  performance  measure  that  is 
to  be  achieved  by  a  specific  calendar  date.  In  the 

VA’s  spreadsheet,  this  is  referred  to  as  “Fully 
Successful.” 

Qualitative  Performance  Measures 


Qualitative  measures  are  also  referred  to  as  categorical  data. 

Qualitative  indicators  are  viewed  as  subjective  that  are  filtered  through 
perceptions.  They  can  therefore  be  imprecise  when  compared  to 
quantitative  measures. 


Document  development 

New  Technology  Introduction 

Customer  Satisfaction 

Plan  completed 

Requirements  defined 

■ 

Excellent 

Research  completed 

Design  completed 

■ 

Very  good 

Draft  complete 

Prototype  developed 

■ 

Good 

Draft  peer-reviewed 

Testing  completed 

■ 

Satisfactory 

Completed/Approved 

Rollout  completed 

■ 

Poor 

Very  poor 
Unacceptable 
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OIA  Performance  Management 


OIA  Goals 


Program  Initiative 

Project 

Project  Activities 


Outcome  #1 

Outcome  #2 

Outcome  #n 

It  is  at  the  project  level  when 
work  is  actually 
accomplished. 

A  project  is  composed  of 
activities. 

Activities  have  outcome(s). 

Outcomes  are  what  we 
measure. 

We  also  measure  progress 
towards  achieving  the 
outcome. 
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Module  Topics 


The  goal  of  performance  management 
What  is  a  “performance  measure”? 

►  What  do  we  measure? 

Step  through  of  review  comments 
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What  Do  We  Measure? 


Project  Activities 


Outcome  #1 

Outcome  #2 

Outcome  #n 

Quality  of  outcome 

Progress  towards 
achieving  outcome 
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What  Do  We  Mean  By  “Quality”? 


In  technical  usage,  quality  can  have  two  meanings: 


the  characteristics  of  a  product  or  service  that  bear  on  its  ability  to 
satisfy  stated  or  implied  needs  (According  to  Philip  Crosby,  quality 
means  “conformance  to  requirements.”) 


Oa  product  or  service  free  of  deficiencies  (According  to  Joseph  Juran, 
quality  means  “fitness  for  use.”) 


Source:  The  American  Society  for  Quality;  Quality  Glossary; 
http://asq.0rg/gl0ssary/q.html 
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Comparing  the  Definitions  of  Quality 


Features  that  meet  or 
exceed  customer  needs 


(  Freedom  from  deficiencies 


Higher  quality  enables  an  organization  to: 

■  increase  customer  satisfaction 

■  meet  or  exceed  competition 

■  increase  market  share 

■  secure  premium  prices 


Higher  quality  enables  an  organization  to: 

■  reduce  rework  and  waste 

■  reduce  field  failures 

■  reduce  customer  dissatisfaction 

■  reduce  pilot  testing  time 

■  shorten  time  to  put  new  products  on  market 

■  increase  capacity 

■  improve  delivery  performance 
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positive 

impact 


HI  Initiative/Activity 


Has  at  least  one 


Has  at  least  one 


( 


Objective 


Outcome 


Intended  to  benefit 


The  best  performance  measures 
evaluate  the  impact  of 
the  outcome  of  the  activity 
on  the  customer. 


Evaluate  impact  from  the 
customer’s  perspective! 


Internal  or  external  customer 
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A  staff  person  in  your  division  has  been  tasked 
to  develop  a  training  course  to  teach  veterans 
how  to  log  onto  an  HI  service. 

Which  of  the  following  performance  measures 
is  the  best ...  and  why? 

1 .  The  number  of  pages  of  materials  prepared  for  the  training  course. 

2.  The  number  of  students  that  have  registered  for  the  training. 

3.  Course  evaluations  -  the  percentage  of  training  course  attendees  that 
rated  the  course  as  “satisfactory”  or  “excellent.” 

4.  The  percentage  of  veterans  that  were  able  to  log  on  to  the  HI  service 
without  help  from  the  call  center  (after  receiving  the  training). 

5.  Post-course  survey  results  of  veterans  that  took  the  training. 
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Key  to  a  Good  a  Outcome-Based  Performance 
Measure 


■  Be  careful  to  distinguish  an  activity  from  the  outcome  of  the 
activity.  Performing  a  task  or  a  series  of  tasks  is  not  an  outcome! 

■  Don’t  write  the  performance  measure  from  your  perspective.  Write 
it  from  the  customer’s  perspective. 

■  Ask  yourself  what  impact  did  the  outcome  (product  or  service) 
have  on  the  user  of  the  service. 

■  Check  and  validate  that  the  performance  measure  provides  insight 
into  whether  you  are  addressing  the  OIA  goal  that  the  activity  is 
mapped  to. 


Software  Engineering  Institute 


Carnegie  Mellon  University 


Performance  Measurement  at  OIT 
Mark  Kasunic,  August,  2014 

©2014  Carnegie  Mellon  University 


27 


What  Do  We  Measure? 


Project  Activities 


Outcome  #1 

Outcome  #2 

Outcome  #n 

Quality  of  outcome 


Progress  towards 
achieving  outcome 
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Why  Measure  Progress? 


Measuring  the  results  of  your  commitment  allows  you  to  look  forward  and 
effectively  plan  next  steps. 


Measuring  schedule  performance  also  provides  an  early  warning  indicator  that 
alerts  you  if  the  project  is  slipping. 


Understanding  schedule  progress 
allows  you  to  take  corrective  actions 
(a  recovery  plan)  so  that  you  meet 
your  commitment. 

Schedule  performance  can  be  measured 
using: 

■  a  quantitative  measure 

or 

■  a  qualitative  measure 
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Quantitative  Example:  Earned  Value 


Cost  Variance 


Estimate  at 
Completion  (EAC) 


Actual  Cost  of  Work 


Variance  at 
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Qualitative  Example:  Milestones  Completed 


MILESTONE  1 


MILESTONE  2 


MILESTONE  3 


PROJECT  END 
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Example:  Schedule-Based  Qualitative 

Performance  Measure 


New  Technology  Introduction  Target  Milestone  Date 


Requirements  defined 
Design  completed 
Prototype  developed 
Testing  completed 
Rollout  completed 


Oct.  22,  2014 
Nov.  30,  2014 
March  1,  2015 
April  1,2015 
August  15,  2015  ~j~ 


Used  to  track  progress 
Fully  Successful  criterion 
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Module  Topics 


The  goal  of  performance  management 
What  is  a  “performance  measure”? 
What  do  we  measure? 

►  Step  through  of  review  comments 
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Expressed  “outcome”  as  something  that 
developers  “do.” 

What  is  the  outcome  of  the  execution  of  the 
processes?  What  is  the  benefit  that  will  be 
realized?  How  will  you  measure  the  benefit? 

How  will  you  know  if  you  have  been  successful? 

Completed  by  beginning  of  Q2?  Or  by  the  end  of 
Q2? 


1 


Measure 


Expressed  as  an  “activity”  but  not  as  a 
measure. 

An  “activity”  column  should  be  added. 

Activity  description  uses  vague.  This 
makes  identifying  a  valid  performance 
measure  difficult. 


Int  l  ease  Al  staff  awareness  of 
local  field  issues, 
considerations,  and  overall 
situation 


w 


Fully  Successful 


Q2:  Develop  and  execute 
processes  for  staff  to  visit 
local  VAMCs 

Conduct  local  site  visits  to 
meet  with  staff  and 
Veterans  at  the  number  of 
sites  per  quarter  below: 

Q3: 1 
Q4:  2 


Expressed  as  an  “activity.” 

What  is  the  impact  of  the  site  visits? 

How  will  you  know  if  you  have  been 
successful? 
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P  Goal# 


The  “hoped-for”  outcome  of  the  activity 
is  to  increase  awareness.  So,  this  is 
the  attribute  that  should  likely  be 
measured. 


What  specifically  is  the  product  or 
service?  Is  the  process  a  “document?” 

I  infer  that  the  intention  is  to  design 
and  document  a  process  that  when 
performed  will  lead  to  increased  staff 
awareness. 


3.a 


Program 

Office 


Al 


Measure 

POC 


Elizabeth 

Johnson 


Measure 

Helper 


Susan 

Bookstaver 


Division  Initiative 


Measure 


Al  organizational 
improvement 


Increase  Al  staff  awareness  of 
local  field  issues, 
considerations,  and  overall 
situation 


These  are  related  to  schedule 
performance. 


Fully  Sue 


Q2:  Develop  and  execute 
processes  for  staff  to  visit 
local  VAMCs 

Conduct  local  site  visits  to 
meet  with  staff  and 
Veterans  at  the  number  of 
sites  per  quarter  below: 

Q3: 1 
Q4:  2 
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Remake 


Activity 

Performance  measure 

Target  (a.k.a.  “Fully  Successful) 

Document  and  deploy  a 
process  that  will  lead  to 
increased  staff 
awareness  of 
local  field  issues. 

Schedule: 

“Process  Status  (PS)”  with 
categorical  variables: 

■  Draft  completed 

■  Draft  peer-reviewed 

■  Process  doc 
completed/approved 

•  Draft  completed  by  10-15-2014 

•  Draft  peer-reviewed”  by  11-05-2014 

•  Process  doc  completed/approved 
by  12-08-2014 

Schedule: 

#  of  sites  where  process  was 
deployed 

1  (by  end  of  Q2) 

2  (by  end  of  Q3) 

Quality/Impact: 

#  of  staff  responses  of  category 
“increased”  or  “significantly 
increased”  to  a  survey  question 
asking  whether  their  awareness 
as  increased 

90%  respond  “increased”  or 
“significantly  increased” 
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Is  the  Mapping  Valid? 
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Checking  Validity 


OIT  Objective: 

3.b  Contracting  -  Improve  OlA’s  ability  to  obtain  contracted 

resources,  by  championing  changes  to  the  current  process  and 
leveraging  innovative  contracting  options  external  to  the  agency. 


Does  the  program  activity 
help  OIT  meet  this 
objective? 


Program  activity: 

Increase  Al  staff  awareness  of  local  field 
issues,  considerations,  and  overall  situation 


Software  Engineering  Institute 


Carnegie  Mellon  University 


Performance  Measurement  at  OIT 
Mark  Kasunic,  August,  2014 

©2014  Carnegie  Mellon  University 


38 


“Accomplishing”  is  an  activity.  It  is  not  a 
measure.  What  will  you  measure  to 
understand 

•  the  degree  of  VHA  office  integration 

•  the  clar 

*ity  of  roles  and  responsibilitie 

3S  s  Measure 

Fully  Successful 

Execute  a  plan  to  Integrate  all 
key  VHA  offices  into  VE  with 
clear  roles  and  responsibilities. 

Accomplish  Integration  Plan 

activities 

Plan  components 
implemented 

The  outcomes  of  this  activity  are: 

1 .  Integrated  VHA  offices 

2.  Development  of  clear  roles  and 
responsibilities 

So,  these  are  the  things  that  should  be 
measured. 


But  how  will  “success”  be  measured? 
What  if  all  the  plan  components  are 
implemented,  but  the  VHA  offices 
remain  unintegrated? 

When  will  the  integration  be 
accomplished? 
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Activity 

Performance  measure 

Target  (a.k.a.  “Fully  Successful) 

Integrate  key  VHA  offices 
into  VE  with  clear  roles 
and  responsibilities 

Schedule: 

“Roles/Responsibilites  doc 
status)”  with  categorical 
variables: 

■  Doc  drafted 

■  Draft  doc  peer-reviewed 

■  Roles/responsibilities  doc 
completed  and  approved 

•  Draft  completed  by  10-15-2014 

•  Draft  peer-reviewed”  by  11-05-2014 

•  Roles/responsibilities  doc 
completed/approved  by  12-08-2014 

Schedule 

[How  will  the  progress  towards 
integration  be  measured?  What 
the  key  milestones?] 

#  of  VHA  offices  integrated  into  VE  by 
date? 

Quality/Impact: 

#  of  staff  responses  of  category 
“clear”  or  “very  clear”  to  a 
survey  question  asking  about 
the  clarity  of  the  roles  and 
responsibilities 

90%  respond  “clear”  or  “very  clear” 
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The  activity  is  unclear.  I  infer  that  this  development  of  a 
training  course?  However,  the  “Fully  Successful”  criterion 
indicates  that  the  training  is  delivered.  Is  the  activity  to 
develop  and  conduct?  Or  just  to  conduct? 


Activity 

Success  Measure 

Fully  Successful 

Complete  MyVeHU  training  for 

A  training  session 

Conduct  one  training  session 

JLV2.2 

by  end  of  Q1 

This  is  not  expressed  as  a  measure. 


This  is  a  schedule-based  performance 
measure. 

How  will  you  assess  the  quality  of  the 
delivered  training?  Did  the  training  have  the 
intended  impact? 
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Example  Remake 


Activity 

Performance  measure 

Target  (a.k.a.  “Fully  Successful) 

Deliver  the  MyVeHU 
training  course  for  JLV 

2.2 

#  of  students  that  successfully 
meet  the  objectives  of  the 
course  (assumes  summative 
evaluation) 

90%  of  registered  students  meet  the 
course  objectives 

#  student  course  ratings  of  90%  of  students  rate  the  course  as 

“effective”  or  “very  effective”  “effective”  or  “very  effective” 


Note:  If  course  development  is  involved,  then  schedule-based  performance 
measures  should  be  developed. 
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Is  the  Mapping  Valid? 
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Checking  Validity 


OIT  Objective: 

3.b  Contracting  -  Improve  OlA’s  ability  to  obtain  contracted 

resources,  by  championing  changes  to  the  current  process  and 
leveraging  innovative  contracting  options  external  to  the  agency. 


Does  the  program  activity 
help  OIT  meet  this 
objective? 


Program  activity: 

Increase  Al  staff  awareness  of  local  field 
issues,  considerations,  and  overall  situation 
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Contact  Information 


Mark  Kasunic 

Senior  Member  of  Technical  Staff 
Software  Engineering  &  Acquisition  Practices 
Software  Solutions  Division 
Telephone:  +1  412-268-5863 
Email:  info@sei.cmu.edu 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 


Web 

www.sei.cmu.edu 

www.sei.cmu.edu/contact.cfm 


Customer  Relations 

Email:  info@sei.cmu.edu 
Telephone:  +1  412-268-5800 

SEI  Phone:  +1  412-268-5800 

SEI  Fax:  +1  412-268-6257 
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